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Abstract 


Discrete Eourier transforms provide a significant speedup in the computation of con¬ 
volutions in deep learning. In this work, we demonstrate that, beyond its advantages 
for efficient computation, the spectral domain also provides a powerful representa¬ 
tion in which to model and train convolutional neural networks (CNNs). 

We employ spectral representations to introduce a number of innovations to CNN 
design. Eirst, we propose spectral pooling, which performs dimensionality reduc¬ 
tion by truncating the representation in the frequency domain. This approach pre¬ 
serves considerably more information per parameter than other pooling strategies 
and enables flexibility in the choice of pooling output dimensionality. This repre¬ 
sentation also enables a new form of stochastic regularization by randomized mod¬ 
ification of resolution. We show that these methods achieve competitive results on 
classification and approximation tasks, without using any dropout or max-pooling. 

Einally, we demonstrate the effectiveness of complex-coefficient spectral parameter¬ 
ization of convolutional filters. While this leaves the underlying model unchanged, 
it results in a representation that greatly facilitates optimization. We observe on a 
variety of popular CNN configurations that this leads to significantly faster conver¬ 
gence during training. 


1 Introduction 

Convolutional neural networks (CNNs) (LeCun et al., 1989) have been used to achieve unparal¬ 
leled results across a variety of benchmark machine learning problems, and have been applied suc¬ 
cessfully throughout science and industry for tasks such as large scale image and video classifica¬ 
tion (Krizhevsky et al., 2012; Karpathy et al., 2014). One of the primary challenges of CNNs, how¬ 
ever, is the computational expense necessary to train them. In particular, the efficient implementation 
of convolutional kernels has been a key ingredient of any successful use of CNNs at scale. 

Due to its efficiency and the potential for amortization of cost, the discrete Eourier transform has long 
been considered by the deep learning community to be a natural approach to fast convolution (Bengio 
& LeCun, 2007). More recently, Mathieu et al. (2013); Vasilache et al. (2014) have demonstrated that 
convolution can be computed significantly faster using discrete Eourier transforms than directly in 
the spatial domain, even for tiny filters. This computational gain arises from the convenient property 
of operator duality between convolution in the spatial domain and element-wise multiplication in the 
frequency domain. 
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In this work, we argue that the frequency domain offers more than a computational trick for convolu¬ 
tion: it also provides a powerful representation for modeling and training CNNs. Frequency decom¬ 
position allows studying an input across its various length-scales of variation, and as such provides a 
natural framework for the analysis of data with spatial coherence. We introduce two applications of 
spectral representations. These contributions can be applied independently of each other. 


Spectral parametrization We propose the idea of learning the filters of CNNs directly in the fre¬ 
quency domain. Namely, we parametrize them as maps of complex numbers, whose discrete Fourier 
transforms correspond to the usual filter representations in the spatial domain. 

Because this mapping corresponds to unitary transformations of the filters, this reparametrization does 
not alter the underlying model. However, we argue that the spectral representation provides an appro¬ 
priate domain for parameter optimization, as the frequency basis captures typical filter structure well. 
More specifically, we show that filters tend to be considerably sparser in their spectral representations, 
thereby reducing the redundancy that appears in spatial domain representations. This provides the 
optimizer with more meaningful axis-aligned directions that can be taken advantage of with standard 
element-wise preconditioning. 

We demonstrate the effectiveness of this reparametrization on a number of CNN optimization tasks, 
converging 2-5 times faster than the standard spatial representation. 


Spectral pooling Pooling refers to dimensionality reduction used in CNNs to impose a capacity 
bottleneck and facilitate computation. We introduce a new approach to pooling we refer to as spectral 
pooling. It performs dimensionality reduction by projecting onto the frequency basis set and then 
truncating the representation. 

This approach alleviates a number of issues present in existing pooling strategies. For example, while 
max pooling is featured in almost every CNN and has had great empirical success, one major criticism 
has been its poor preservation of information (Hinton, 2014b,a). This weakness is exhibited in two 
ways. First, along with other stride-based pooling approaches, it implies a very sharp dimensionality 
reduction by at least a factor of 4 every time it is applied on two-dimensional inputs. Moreover, while 
it encourages translational invariance, it does not utilize its capacity well to reduce approximation loss: 
the maximum value in each window only refiects very local information, and often does not represent 
well the contents of the window. 

In contrast, we show that spectral pooling preserves considerably more information for the same 
number of parameters. It achieves this by exploiting the non-uniformity of typical inputs in their 
signal-to-noise ratio as a function of frequency. For example, natural images are known to have an 
expected power spectrum that follows an inverse power law: power is heavily concentrated in the 
lower frequencies — while higher frequencies tend to encode noise (Torralba & Oliva, 2003). As 
such, the elimination of higher frequencies in spectral pooling not only does minimal damage to the 
information in the input, but can even be viewed as a type of denoising. 

In addition, spectral pooling allows us to specify any arbitrary output map dimensionality. This per¬ 
mits reduction of the map dimensionality in a slow and controlled manner as a function of network 
depth. Also, since truncation of the frequency representation exactly corresponds to reduction in res¬ 
olution, we can supplement spectral pooling with stochastic regularization in the form of randomized 
resolution. 

Spectral pooling can be implemented at a negligible additional computational cost in convolutional 
neural networks that employ FFT for convolution kernels, as it only requires matrix truncation. We 
also note that these two ideas are both compatible with the recently-introduced method of batch nor¬ 
malization (Ioffe & Szegedy, 2015), permitting even better training efficiency. 


2 The Discrete Fourier Transform 


The discrete Fourier transform (DFT) is a powerful way to decompose a spatiotemporal signal. In this 
section, we provide an introduction to a number of components of the DFT drawn upon in this work. 
We confine ourselves to the two-dimensional DFT, although all properties and results presented can 
be easily extended to other input dimensions. 
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(a) DFT basis functions. 



Figure 1: Properties of discrete Fourier transforms, (a) All discrete Fourier basis functions of map 
size 8x8. Note the equivalence of some of these due to conjugate symmetry, (b) Examples of input 
images and their frequency representations, presented as log-amplitudes. The frequency maps have 
been shifted to center the DC component. Rays in the frequency domain correspond to spatial domain 
edges aligned perpendicular to these, (c) Conjugate symmetry patterns for inputs with odd (top) and 
even (bottom) dimensionalities. Orange: real-valuedness constraint. Blue: no constraint. Gray: 
value fixed by conjugate symmetry. 


Given an input x G (we address the constraint of real inputs in Subsection 2.1), its 2D 

DFT ^(x) G is given by 


M-lAT-l 

Vh e {0,..., M - l},Vw e {0,..., N - 1} . 

V m=0 n=0 

The DFT is linear and unitary, and so its inverse transform is given by = ^(.) *, namely the 

conjugate of the transform itself. 

Intuitively, the DFT coefficients resulting from projections onto the different frequencies can be 
thought of as measures of correlation of the input with basis functions of various length-scales. See 
Figure 1(a) for a visualization of the DFT basis functions, and Figure 1(b) for examples of input- 
frequency map pairs. 

The widespread deployment of the DFT can be partially attributed to the development of the Fast 
Fourier Transform (FFT), a mainstay of signal processing and a standard component of most math li¬ 
braries. The FFT is an efficient implementation of the DFT with time complexity O {MN log (MN)). 

Convolution using DFT One powerful property of frequency analysis is the operator duality be¬ 
tween convolution in the spatial domain and element-wise multiplication in the spectral domain. 
Namely, given two inputs x, f G we may write 

^(x * f) = ^(x) 0 ^(f) (1) 

where by * we denote a convolution and by 0 an element-wise product. 

Approximation error The unitarity of the Fourier basis makes it convenient for the analysis of 
approximation loss. More specifically, Parseval’s Theorem links the £2 loss between any input x and 
its approximation x to the corresponding loss in the frequency domain: 

||x-x||^ = ll^(x)-^(x)ll^ . (2) 

An equivalent statement also holds for the inverse DFT operator. This allows us to quickly assess how 
an input is affected by any distortion we might make to its frequency representation. 

2.1 Conjugate symmetry constraints 

In the following sections of the paper, we will propagate signals and their gradients through DFT and 
inverse DFT layers. In these layers, we will represent the frequency domain in the complex field. 
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However, for all layers apart from these, we would like to ensure that both the signal and its gradient 
are constrained to the reals. A necessary and sufficient condition to achieve this is conjugate symmetry 
in the frequency domain. Namely, for any transform y = ^(x) of some input x, it must hold that 

Ymn = y(M-m)modM,(Ar-n)modAr G {0, . . . , M — 1}, Vn G {0, . . . , — 1} . (3) 

Thus, intuitively, given the left half of our frequency map, the diminished number of degrees of free¬ 
dom allows us to reconstruct the right. In effect, this allows us to store approximately half the pa¬ 
rameters that would otherwise be necessary. Note, however, that this does not reduce the effective 
dimensionality, since each element consists of real and imaginary components. The conjugate sym¬ 
metry constraints are visualized in Figure 1(c). Given a real input, its DFT will necessarily meet these. 
This symmetry can be observed in the frequency representations of the examples in Figure 1(b). How¬ 
ever, since we seek to optimize over parameters embedded directly in the frequency domain, we need 
to pay close attention to ensure the conjugate symmetry constraints are enforced upon inversion back 
to the spatial domain (see Subsection 2.2). 


2.2 Differentiation 


Here we discuss how to propagate the gradient through a Fourier transform layer. This analysis can 
be similarly applied to the inverse DFT layer. Define x G and y = ^(x) to be the input 

and output of a DFT layer respectively, and R : -> M a real-valued loss function applied to y 

which can be considered as the remainder of the forward pass. Since the DFT is a linear operator, 
its gradient is simply the transformation matrix itself. During back-propagation, then, this gradient is 
conjugated, and this, by DFT unitarity, corresponds to the application of the inverse transform: 


= . 
9x \dy) 


(4) 


There is an intricacy that makes matters a bit more complicated. Namely, the conjugate symme¬ 
try condition discussed in Subsection 2.1 introduces redundancy. Inspecting the conjugate symme¬ 
try constraints in Equation (3), we note their enforcement of the special case yoo G M for odd, 
and yoo, yiv o, Yo ^^ G M for A^ even. For all other indices they enforce conjugate equality of 
pairs of distinct elements. These conditions imply that the number of unconstrained parameters is 
about half the map in its entirety. 


3 Spectral Pooling 

The choice of a pooling technique boils down to the selection of an appropriate set of basis func¬ 
tions to project onto, and some truncation of this representation to establish a lower-dimensionality 
approximation to the original input. The idea behind spectral pooling stems from the observation that 
the frequency domain provides an ideal basis for inputs with spatial structure. We first discuss the 
technical details of this approach, and then its advantages. 

Spectral pooling is straightforward to understand and to implement. We assume we are given an in¬ 
put X G and some desired output map dimensionality H x W. First, we compute the discrete 

Fourier transform of the input into the frequency domain as y = ^(x) G and assume that 

the DC component has been shifted to the center of the domain as is standard practice. We then crop 
the frequency representation by maintaining only the central H x W submatrix of frequencies, which 
we denote as y G Finally, we map this approximation back into the spatial domain by taking 


Algorithm 1: Spectral pooling 

Input: Map x G output size H xW 

Output: Pooled map x G 
1: y ^ ^(x) 

2: y ^ CROPSPECTRUM(y,Ff x W) 

3: y ^ TREATCORNERCASES(y) 

4: x^^-i(y) 


Algorithm 2: Spectral pooling back-propagation 

Input: Gradient w.r.t output ^ 

Output: Gradient w.r.t input 

2: z ^ RemoveRedundancy(z) 

3: z ^ PadSpectrum(z, M x N ) 

4: z ^ RecoverMap(z) 

5: If 
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Figure 2: Approximations for different pooling schemes, for different factors of dimensionality re¬ 
duction. Spectral pooling projects onto the Fourier basis and truncates it as desired. This retains 
significantly more information and permits the selection of any arbitrary output map dimensionality. 


its inverse DFT as x = G These steps are listed in Algorithm 1. Note that some of 

the conjugate symmetry special cases described in Subsection 2.2 might be broken by this truncation. 
As such, to ensure that x is real-valued, we must treat these individually with TreatCornerCases, 
which can be found in the supplementary material. 

Figure 2 demonstrates the effect of this pooling for various choices of H x W. The back- 
propagation procedure is quite intuitive, and can be found in Algorithm 2 (RemoveRedundancy 
and RecoverMap can be found in the supplementary material). In Subsection 2.2, we addressed the 
nuances of differentiating through DFT and inverse DFT layers. Apart from these, the last component 
left undiscussed is differentiation through the truncation of the frequency matrix, but this corresponds 
to a simple zero-padding of the gradient maps to the appropriate dimensions. 

In practice, the DFTs are the computational bottlenecks of spectral pooling. However, we note that 
in convolutional neural networks that employ FFTs for convolution computation, spectral pooling can 
be implemented at a negligible additional computational cost, since the DFT is performed regardless. 

We proceed to discuss a number of properties of spectral pooling, which we then test comprehensively 
in Section 5. 


3.1 Information preservation 


Spectral pooling can significantly increase the amount of retained information relative to max-pooling 
in two distinct ways. First, its representation maintains more information for the same number of 
degrees of freedom. Spectral pooling reduces the information capacity by tuning the resolution of the 
input precisely to match the desired output dimensionality. This operation can also be viewed as linear 
low-pass filtering and it exploits the non-uniformity of the spectral density of the data with respect to 
frequency. That is, that the power spectra of inputs with spatial structure, such as natural images, 
carry most of their mass on lower frequencies. As such, since the amplitudes of the higher frequencies 
tend to be small, Parseval’s theorem from Section 2 informs us that their elimination will result in a 
representation that minimizes the £2 distortion after reconstruction. 

Second, spectral pooling does not suffer from the sharp reduction in output dimensionality exhibited 
by other pooling techniques. More specifically, for stride-based pooling strategies such as max pool¬ 
ing, the number of degrees of freedom of two-dimensional inputs is reduced by at least 75% as a 
function of stride. In contrast, spectral pooling allows us to specify any arbitrary output dimensional¬ 
ity, and thus allows us to reduce the map size gradually as a function of layer. 
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(a) Filters over time. 



(b) Sparsity patterns. (c) Momenta distributions. 


Figure 3: Learning dynamics of CNNs with spectral parametrization. The histograms have been 
produced after 10 epochs of training on CIFAR-10 by each method, but are similar throughout, (a) 
Progression over several epochs of filters parametrized in the frequency domain. Each pair of columns 
corresponds to the spectral parametrization of a filter and its inverse transform to the spatial domain. 
Filter representations tend to be more local in the Fourier basis, (b) Sparsity patterns for the different 
parametrizations. Spectral representations tend to be considerably sparser, (c) Distributions of mo¬ 
menta across parameters for CNNs trained with and without spectral parametrization. In the spectral 
parametrization considerably fewer parameters are updated. 


3.2 Regularization via resolution corruption 

We note that the low-pass filtering radii, say Rh and Rw, can be chosen to be smaller than the output 
map dimensionalities Ff, W. Namely, while we truncate our input frequency map to size H x W, 
we can further zero-out all frequencies outside the central Rh x Rw square. While this maintains 
the output dimensionality H x W of the input domain after applying the inverse DFT, it effectively 
reduces the resolution of the output. This can be seen in Figure 2. 

This allows us to introduce regularization in the form of random resolution reduction. We apply this 
stochastically by assigning a distribution pr{') on the frequency truncation radius (for simplicity we 
apply the same truncation on both axes), sampling from this a random radius at each iteration, and 
wiping out all frequencies outside the square of that size. Note that this can be regarded as an appli¬ 
cation of nested dropout (Rippel et al., 2014) on both dimensions of the frequency decomposition of 
our input. In practice, we have had success choosing Pr{') = (')’ ^ uniform distribution 

stretching from some minimum value all the way up to the highest possible resolution. 


4 Spectral Parametrization of CNNs 

Here we demonstrate how to learn the filters of CNNs directly in their frequency domain represen¬ 
tations. This offers significant advantages over the traditional spatial representation, which we show 
empirically in Section 5. 

Let us assume that for some layer of our convolutional neural network we seek to learn filters 
of size H X W. To do this, we parametrize each filter f G in our network directly in 

the frequency domain. To attain its spatial representation, we simply compute its inverse DFT 
as ^“^(f) G From this point on, we proceed as we would for any standard CNN by com¬ 

puting the convolution of the filter with inputs in our mini-batch, and so on. 

The back-propagation through the inverse DFT is virtually identical to the one of spectral pooling 
described in Section 3. We compute the gradient as outlined in Subsection 2.2, being careful to obey 
the conjugate symmetry constraints discussed in Subsection 2.1. 

We emphasize that this approach does not change the underlying CNN model in any way — only the 
way in which it is parametrized. Hence, this only affects the way the solution space is explored by the 
optimization procedure. 

4.1 Leveraging filter structure 

This idea exploits the observation that CNN filters have a very characteristic structure that reappears 
across data sets and problem domains. That is, CNN weights can typically be captured with a small 
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Method 

CIFAR-10 

CIFAR-100 

Stochastic pooling 

15.13% 

41.51% 

Maxout 

11.68% 

38.57% 

Network-in-network 

10.41% 

35.68% 

Deeply supervised 

9.78% 

34.57% 

Spectral pooling 

8.6% 

31.6% 


(b) Classification rates. 


Figure 4: (a) Average information dissipation for the ImageNet validation set as a function of fraction 
of parameters kept. This is measured in ^2 error normalized by the input norm. The red horizontal 
line indicates the best error rate achievable by max pooling, (b) Test errors on CIFAR-10/100 without 
data augmentation of the optimal spectral pooling architecture, as compared to current state-of-the-art 
approaches: stochastic pooling (Zeiler & Fergus, 2013), Maxout (Goodfellow et al., 2013), network- 
in-network (Lin et al., 2013), and deeply-supervised nets (Lee et al., 2014). 


number of degrees of freedom. Represented in the spatial domain, however, this results in significant 
redundancy. 

The frequency domain, on the other hand, provides an appealing basis for filter representation: char¬ 
acteristic filters (e.g., Gabor filters) are often very localized in their spectral representations. This 
follows from the observation that filters tend to feature very specific length-scales and orientations. 
Hence, they tend to have nonzero support in a narrow set of frequency components. This hypothesis 
can be observed qualitatively in Figure 3(a) and quantitatively in Figure 3(b). 

Empirically, in Section 5 we observe that spectral representations of filters leads to a convergence 
speedup by 2-5 times. We remark that, had we trained our network with standard stochastic gradient 
descent, the linearity of differentiation and parameter update would have resulted in exactly the same 
filters regardless of whether they were represented in the spatial or frequency domain during training 
(this is true for any invertible linear transformation of the parameter space). 

However, as discussed, this parametrization corresponds to a rotation to a more meaningful axis align¬ 
ment, where the number of relevant elements has been significantly reduced. Since modern optimizers 
implement update rules that consist of adaptive element-wise rescaling, they are able to leverage this 
axis alignment by making large updates to a small number of elements. This can be seen quantitatively 
in Figure 3(c), where the optimizer — Adam (Kingma & Ba, 2015), in this case — only touches a 
small number of elements in its updates. 

There exist a number of extensions of the above approach we believe would be quite promising in 
future work; we elaborate on these in the discussion. 


5 Experiments 

We demonstrate the effectiveness of spectral representations in a number of different experiments. We 
ran all experiments on code optimized for the Xeon Phi coprocessor. We used Spearmint (Snoek et al., 
2015) for Bayesian optimization of hyperparameters with 5-20 concurrent evaluations. 

5.1 Spectral pooling 

Information preservation We test the information retainment properties of spectral pooling on the 
validation set of ImageNet (Russakovsky et al., 2015). For the different pooling strategies we plot 
the average approximation loss resulting from pooling to different dimensionalities. This can be seen 
in Figure 4. We observe the two aspects discussed in Subsection 3.1: first, spectral pooling permits 
significantly better reconstruction for the same number of parameters. Second, for max pooling, the 
only knob controlling the coarseness of approximation is the stride, which results in severe quantiza¬ 
tion and a constraining lower bound on preserved information (marked in the figure as a horizontal 
red line). In contrast, spectral pooling permits the selection of any output dimensionality, thereby 
producing a smooth curve over all frequency truncation choices. 
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(a) Training curves. 


Architecture 

Filter 

size 

Speedup 

factor 

Deep (7) 

3x3 

2.2 

Deep (7) 

5x5 

4.8 

Generic ( 6 ) 

3x3 

2.2 

Generic ( 6 ) 

5x5 

5.1 

Sp. Pooling (5) 

3x3 

2.4 

Sp. Pooling (5) 

5x5 

4.8 


(b) Speedup factors. 


Figure 5: Optimization of CNNs via spectral parametrization. All experiments include data augmen¬ 
tation. (a) Training curves for the various experiments. The remainder of the optimization past the 
matching point is marked in light blue. The red diamonds indicate the relative epochs in which the 
asymptotic error rate of the spatial approach is achieved, (b) Speedup factors for different architec¬ 
tures and filter sizes. A non-negligible speedup is observed even for tiny 3x3 filters. 


Classification with convolutional neural networks We test spectral pooling on different classifi¬ 
cation tasks. We hyperparametrize and optimize the following CNN architecture: 


(C 


96-|-32m 

3x3 


SP 


X [rfHm\) 


M 

m=l 


^96+32M ^10/100 

'^Ixl ^ '^Ixl 


GA ^ Softmax (5) 


Here, by Cf we denote a convolutional layer with F filters each of size S, by a spectral pool¬ 
ing layer with output dimensionality S, and GA the global averaging layer described in Lin et al. 
(2013). We upper-bound the number of filters per layer as 288. Every convolution and pooling 
layer is followed by a ReLU nonlinearity. We let Hm be the height of the map of layer m. Hence, 
each spectral pooling layer reduces each output map dimension by factor 7 G (0,1). We assign fre¬ 
quency dropout distribution m, a, /3) = (') layer m, total layers M and with 

c^(a, 13) = a — a) for some constants a, G M. This parametrization can be thought of as 

some linear parametrization of the dropout rate as a function of the layer. 


We perform hyperparameter optimization on the dimensionality decay rate 7 G [0.25, 0.85], number 
of layers Mg {!,...,15}, resolution randomization hyperparameters a, /3 G [0, 0.8], weight decay 
rate in [10“^, 10“^], momentum in [1 — 0.1^’^, 1 — 0.1^] and initial learning rate in [0.1^, 0.1]. We 
train each model for 150 epochs and anneal the learning rate by a factor of 10 at epochs 100 and 140. 
We intentionally use no dropout nor data augmentation, as these introduce a number of additional 
hyperparameters which we want to disambiguate as alternative factors for success. 


Perhaps unsurprisingly, the optimal hyperparameter configuration assigns the slowest possible 
layer map decay rate 7 = 0.85. It selects randomized resolution reduction constants of about 
a ^ 0.30, /3 « 0.15, momentum of about 0.95 and initial learning rate 0.0088. These settings allow 
us to attain classification rates of 8 . 6 % on CIFAR-10 and 31.6% on CIFAR-100. These are competi¬ 
tive results among approaches that do not employ data augmentation: a comparison to state-of-the-art 
approaches from the literature can be found in Table 4(b). 


5.2 Spectral parametrization of CNNs 

We demonstrate the effectiveness of spectral parametrization on a number of CNN optimization tasks, 
for different architectures and for different filter sizes. We use the notation MPj to denote a max 
pooling layer with size S and stride T, and FC^ is a fully-connected layer with F filters. 

The first architecture is the generic one used in a variety of deep learning papers, such as Krizhevsky 
et al. (2012); Snoek et al. (2012); Krizhevsky (2009); Kingma & Ba (2015): 

^3x3 ^ MP^x3 ^ ^ 1 ^X 3 MP^x 3 ^ ^ FC^^^ ^ Softmax ( 6 ) 


The second architecture we consider is the one employed in Snoek et al. (2015), which was shown to 
attain competitive classification rates. It is deeper and more complex: 


96 V ^96 , 2 

^3x3 ^ ^3x3 ^ mP3x3 


192 

^3x3 


^ c. 


192 

3x3 


192 

*^3x3 


mP3x3 


.192 

-1x1 


10/100 

%xl 


GA —y Softmax (7) 
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The third architecture considered is the spectral pooling network from Equation 5. To increase the 
difficulty of optimization and reflect real training conditions, we supplemented all networks with 
considerable data augmentation in the form of translations, horizontal reflections, HSV perturbations 
and dropout. 

We initialized both spatial and spectral filters in the spatial domain as the same values; for the spectral 
parametrization experiments we then computed the Fourier transform of these to attain their frequency 
representations. We optimized all networks using the Adam (Kingma & Ba, 2015) update rule, a 
variant of RMSprop that we find to be a fast and robust optimizer. 

The training curves can be found in Figure 5(a) and the respective factors of convergence speedup in 
Table 5. Surprisingly, we observe non-negligible speedup even for tiny filters of size 3x3, where we 
did not expect the frequency representation to have much room to exploit spatial structure. 

6 Discussion and remaining open problems 

In this work, we demonstrated that spectral representations provide a rich spectrum of applications. 
We introduced spectral pooling, which allows pooling to any desired output dimensionality while 
retaining significantly more information than other pooling approaches. In addition, we showed that 
the Fourier functions provide a suitable basis for filter parametrization, as demonstrated by faster 
convergence of the optimization procedure. 

One possible future line of work is to embed the network in its entirety in the frequency domain. In 
models that employ Fourier transforms to compute convolutions, at every convolutional layer the input 
is FFT-ed and the elementwise multiplication output is then inverse FFT-ed. These back-and-forth 
transformations are very computationally intensive, and as such it would be desirable to strictly remain 
in the frequency domain. However, the reason for these repeated transformations is the application of 
nonlinearities in the forward domain: if one were to propose a sensible nonlinearity in the frequency 
domain, this would spare us from the incessant domain switching. 

In addition, one significant downfall of the DFT approach is its difficulty in handling finite impulse 
response filtering. In particular, its projection onto the various frequencies involves global sums over 
the entire input. Hence, the input domain has perfect spatial locality and no spectral locality, while 
the Fourier domain has perfect spectral locality and no spatial locality. An intermediate solution 
we believe would be very effective is employing wavelets, which provide a middle ground between 
the two approaches. While wavelets have been employed throughout machine learning with great 
promise (Bruna & Mallat, 2013; Oyallon et al., 2013), to our knowledge they have not been used in 
an adaptive way to learn CNNs. 
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